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Abstract —In this letter, we propose a new symbol detection 
method for faster-than-Nyquist signaling (FTNS) systems. Based 
on frame theory, we formulate a symbol detection problem as a 
under-determined linear equation on a finite set. The problem is 
reformulated as a sum-of-absolute-values (SOAV) optimization 
that can be efficiently solved by the fast iterative shrinkage 
thresholding algorithm (FISTA). The proximity operator for the 
convex optimization is derived analytically. Simulation results are 
given to show that the proposed method can successfully detect 
symbols in faster-than-Nyquist signaling systems and has lower 
complexity in terms of computation time. 

Index Terms —Faster-than-Nyquist signaling, Weyl-Helsenberg 
frames, symbol detection, sum of absolute values, fast iterative 
shrinkage thresholding algorithm. 

I. Introduction 

ASTER-than-Nyquist signaling (FTNS), which was pro¬ 
posed by Mazo in 1975, is a framework to transmit signals 
exceeding the Nyquist rate fT]. It is shown that a 24.7 % faster 
symbol rate than the Nyquist rate can be achieved without 
performance loss in terms of the minimum Euclidian distance 
for binary symbols and sine pulses. Moreover, the capacity 
of FTNS is higher than conventional Nyquist rate signaling 
from an information theoretic viewpoint ||2l. For these reasons, 
FTNS has been drawing great attention as a new method 
realizing high-speed data transmission 13. 

Nyquist discovered that if the symbol period is shorter than 
the inverse of the pulse bandwidth, then inter-symbol interfer¬ 
ence (ISI) is unavoidable ||4l- The key idea of actualization 
of FTNS is that, by appropriate signal processing, one can 
sufficiently reduce ISI caused by a shorter symbol period 
than that of the Nyquist criterion requires. When ISI happens 
due to the fast symbol rate, the sifted pulses become linearly 
dependent and it is impossible to recover arbitrary symbols 
in the real axis or the complex plane. However, as pointed 
out in 13, considering the fact that candidates of transmitted 
symbols are elements of a finite set in digital communication, 
we might perfectly reconstruct symbols even when the symbol 
rate is higher than the Nyquist rate. This desirable situation 
can be attained when transmission pulses constitute WeyT 
Heisenberg frames and consequently, we can realize no loss 
FTNS using frame theory 0. Furthermore, it is shown that 
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a system with frames is more robust against doubly selective 
fading than a system with conventional orthogonal pulses 0, 
Q. Because of these merits, we investigate the frame-based 
FTNS 0 in this letter. 

In practice, symbol detection is one of the fundamental 
issues to realize FTNS. As standard detection methods for 
conventional FTNS, maximum likelihood (ML) approaches, 
such as the Viterbi algorithm (VA) and the Bahl-Cocke- 
Jelinek-Raviv (BCJR) algorithm, have been proposed for un¬ 
coded FTNS 0. For coded FTNS, turbo receivers have been 
proposed in ifTOl . One the other hand, as a detection scheme for 
the frame-based FTNS, the I°° minimization method has been 
proposed and it is shown that the computational complexity 
of the method is lower than that of the above algorithms Ha. 
However, further reduction of the complexity will be required 
for the application to practical systems. 

In this letter, we propose a symbol detector for frame-based 
FTNS using sum of absolute values (SOAV) optimization. 
SOAV has been recently proposed as a discrete valued signal 
reconstruction method and is known to be very effective when 
the number of candidate discrete values is not so large m. 
While only noiseless cases are considered in lfT3 . we extend 
the problem formulation into noisy cases. The fast iterative 
shrinkage thresholding algorithm (FISTA) 03 is utilized as 
a solver for the optimization problem, since it requires much 
lower computational complexity. To derive an algorithm based 
on FISTA, we give the closed form of a proximity operator 
for the SOAV optimization. 

The remainder of this letter is as follows: SectionlUgives the 
system model considered in this letter and Sectionlllllproposes 
a symbol detection scheme. In Section|IV]we introduce FISTA 
and derive an associated proximity operator. Simulation results 
are shown in Section [V] to illustrate the effectiveness of the 
proposed method. Section IVTl draws conclusions. 

II. System model 

In this section, we explain the frame-based FTNS system 
considered in this letter. 

Let us consider the following linear modulation: 

N 

x(f) - y ^ Xji (t) , 

n—1 

where N € N is the number of modulation waveforms, 
{xn}n=i £ {-fl,—1}^ are independent identically distributed 
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(i.i.d.) binary symbols, and are the modulation pulses 

whose frequency bandwidth is limited to W. The signal 
through an additive white Gaussian noise (AWGN) channel 
is 

p(t) = x(t) + w(t) 

N 

= Xnhnit) + w{t), 

n—1 

where w{t) is the AWGN with zero mean and power spectral 
density Nq. We assume that the symbol period is set to be T. 
Let M be the dimension of the time-frequency space occupied 
by x{t). {(t>i}^i is set to be an orthonormal basis for such 
a time-frequency space. Define ym = {y{t),4>m{t)),hm,n = 
{hn{t),(j)m{t)), and Wm = form = 

and n = 1,..., W, where (•, •) denotes an inner product. Then 
we obtain the following linear equation HD: 


y = Hx + w, 


( 1 ) 


where x = [xi,... ,xnV ,y = [yi,... ,yM]^, H = 
w = [wi,... ,wm]^ , and represents the transpose, w 
is a zero mean Gaussian random vector with covariance 
Elww^} = (Ao/2)/m, where E{-} and Im denote the 
expectation and the M dimension identity matrix respectively. 
It is assumed that hn is generated by i.i.d variables with zero 
mean and variance 1/M. Then, the modulation matrix H is 
modeled as a random matrix M- 

Note that, for a conventional FTNS system using an or¬ 
dinary matched-filter, H will be a Toeplitz convolution ma¬ 
trix, while discrete valued signal reconstruction scheme with 
SOAV does not work well for such a structured matrix. 
Accordingly, we can also find one of merits of frame-based 
FTNS system. Note also that, we can employ not only binary 
phase shift keying (BPSK) but also quadrature phase shift 
keying (QPSK) for the baseband modulation. When QPSK 
is used, let x be modulated symbols, w be complex-valued 
noise and H he a complex-valued modulation matrix. Define 
X = [Re{i^}, Im{i^}]^, w = [Rejtl;^}, and 




Re{H} -Im{M} ' 
Im{M} Re{iT} J ’ 


where Re{ } and Im{ } denote the real part and the imaginary 
part respectively. Then we obtain the above system O. 


III. Symbol detection via SOAV optimization 

When the system employs faster-than-Nyquist signaling, 
i.e., TW > 1, then M < N. In a noiseless case, the original 
signal X can be found from the constrained under-determined 
system: 

Hz = y, s.t. z G {-bl, -1}^, 

since the number of the symbol set is finite HD. Similarly, in 
a noisy case the problem can be reformulated as 

min || 2 /-Mz|| 2 , (2) 

where || • ||2 represents the norm of the vector. These 
problems, however, have a combinatorial nature and the com¬ 
putation time becomes exponential. 


To reasonably obtain the solution of (|2|i, the i°°- 
minimization method has been proposed for binary symbol 
recovery HD- In a noisy case, this method considers the 
relaxed problem 

min ||z||oo s.t. \\y-Hz\\l < 

where || • ||oo is defined as the i°° norm of the vector. It 
can be solved by repeating the Newton’s method and the 
solution approximates the solution of dD. The computational 
complexity is less than Viterbi algorithm; nevertheless it is 
insufficient for practical needs. 

To tackle with this difficulty, we consider to estimate x by 
solving the following optimization problem: 

+ + (3) 

s.t. \\y-Hz\\l < e^, 

where || • ||i is the norm of the vector, e G M, and In is the 
N dimension vector whose all components are 1. This problem 
formulation can be considered as a noisy version of the SOAV 
optimization considered in the reference Ha. The problem is 
a convex optimization problem and can be efficiently solved 
by several algorithms. In particular, we propose a proximal 
algorithm that has low computational complexity in the next 
section. 

We can interpret Q as follows: Because the original signal 
x has elements of {-bl,—1}, it is expected that about half 
elements of a; — 1 at and a: -b Iat are all zero, if -b 1 and — 1 
appear with equal probability in x. Consequently, based on 
the idea of compressed sensing, we anticipate that the original 
signal can be obtained by finding a vector z which satisfies 
the constraint and makes the norms of z — Iat and z -b Iat 
small. 


IV. Optimization algorithm 

It is well known that the proximal algorithms, which are 
for solving convex optimization problems, have low computa¬ 
tional complexity na. In order to apply a proximal algorithm 
to the optimization problem of Q, we give the closed form 
of a proximity operator for the problem in this section. 

First, we reformulate Q as the unconstrained optimization 
problem 

m^n Ally - Mz||^ + 

where A is a positive number. Note that for any e there exists 
A such that the solution of (IHi becomes equal to the solution 
of OJ. Letting /(z) = X\\y-Hz\\l and g{z) = i||z-lAr||i + 
2 l|z:-b IaIIi, we rewrite the problem as 

min /{ 2 ) + 5 (z). 

zGK.’^ 

Here / is convex and differentiable. Thus if we have the 
proximity operator of g, then FISTA can be applied to the 
problem |[T3, ifTbl . 

Define the proximity operator of g as prox^ (z) = 

argmin„gRN {g{u) -b |||z — u\W'\. Then we have the follow¬ 
ing proposition: 
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Proposition 1: Let ^ : K -A R be 


a + 1 

if a < —2, 

-1 

if -2 < a < -1, 

a 

if —1 < a < 1, 

1 

if 1 < a < 2, 

a — 1 

if 2 < a 


(see also Fig. [T]). Then we have 

prOXg(2:) = 


where Zi is the i-th element of 2 ;. 

Proof: The function g can be written as 

1 ^ 

= 2 

^ i=l 

where Zi is the i-th element of z. From this, we have 

prox {z) = argmin < 

“SR" [j 

R{a, /3) = |a — 1| + |a + 1| + 1/3 — ap, 

where a,/3 S M. It is clear that prox^(z) is obtained by 
minimizing each R{ui,Zi) independently. Thus, we consider 
minimization of R{a, /3) with a G R for fixed /3 G R. 

If a < —1, then we have 

R{a, p) = {a-{P + 1)}2 + 2/3 - 1 =: Ri{a, /3). 

If — 1 < a < 1, then we have 

R{a, P) = {a- Pf + 2 =: i? 2 (a, /?)■ 


^ ^ Zi) > , 


If a> 1, then we have 


i?(a,/3) = {a-(/3-l)}2 

-2/3-l=:i?3(a,/3) 

In summary, we have 


( Ri(^a,/3), 

if a G (— 00 , —1], 

R{a,l3) = < R2{a,l3), 

if a G [-1,1], 

[ R3{a,l3), 

if a G [1, 00 ). 


Based on this, we calculate the proximity operator. We 
consider the cases: 

1) /3 < -2, 

2) -2 < /3 < -1, 


3) -1 < /3 < 1, 

4) 1 < /3 < 2, 

5) 2</3. 

When /3 < — 2, we have 

argmm„g(_^__i]{i?i(a, /3)} = /3 + 1, 

argmin„g[o_i]{i?2(a,/3)} = -1, 
argmin„g[i_^){i?3(a,/3)} = 1. 

Since i?i(/3 + 1,/3) < i? 2 (—1,/?) < 3?3(1,/3), we have 


argmiii{i?(a,/3)} = /3 + 1. 

aeR 


In a similar way, the optimal value can be got in each case. 


{prox {z)}i 


tain the 

proximity operator of g: 

Zi + 1 

if Zi < —2, 

1 - 1 

1 

if —2 < Zi < —1, 


V 
if 

VI 

I 

1 

if 1 < Zi < 2, 

Zi -1 

if 2 < Zi- 


□ 


With the proximity operator, the following algorithm is 
FISTA to solve (01): 

Algorithm 1 (FISTA iUil/).- Fix z^^'l G R^,fi = 1 and L G 
R which is greater than or equal to a Lipshitz constant of V/. 
For fc > 1, 


z(k) 

' ffe+i 
^(/c) 


prox^g 




1 +V^T+d^ 


2 



It is known that z^^'^ converges to a solution of the opti¬ 
mization problem. Note that here V f{z) can be calculated as 
2XH^{Hz — y). Finally, with the obtained optimal solution 
z* we estimate the original signal by 


Xest = sign(z*), 

where sign( ) is the signum function. 


V. Simulation results 

In this section, we give some simulation results to demon¬ 
strate the performance of the proposed method. 

In the simulations we employ QPSK for digital modu¬ 
lation. We model the components of a modulation matrix 
H G as i.i.d. Gaussian variables with zero mean and 

variance 1/M. It is assumed that the elements of the symbol 
vector X are independently and uniformly distributed with 
equal probability. We set \,L, and to be 0.01,0.1, and 
1 M respectively. The maximum number of iterations of FISTA 
is 100. 

Fig. |2] and Fig. [3 show the bit error rate (BER) performance 
against signal-to-noise ratio (SNR) when = 15, M = 10 
and N = 150, M = 100 respectively. That is, the dimensions 
of the real valued matrix H become 30 x 20 and 300 x 200. 
The solid lines represent the performances of the proposed 
detector and the broken lines represent the performances of the 
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Fig. 2. BER performance of the proposed detector (solid) and the 
minimization detector (broken) when A^ = 15,M = 10. 



Fig. 3. BER performance of the proposed detector (solid) and the 
minimization detector (broken) when N = 150, M = 100. 

minimization detector ini. BER performance is obtained 
by averaging BERs for 1000 realizations of the modulation 
matrix for each SNR with the transmission of 900 symbols 
for each realization. Erom these figures we can see that, while 
the performance of the proposed method is slightly better 
than the minimization method with the small modulation 
matrix, considerable performance gain can be achieved with 
the proposed scheme for the large H. 

Next, we compare the proposed method using EISTA with 
the £°° minimization method using the Newton’s method in 
terms of computation time ini. Table U shows the average 
computation times to solve one optimization problem by the 
methods when N = 150, M = 100. In the simulations, we use 
a computer with Intel Core 15-4590 CPU. The results show 
that the computational complexity of the proposed scheme is 
lower than the conventional scheme, while achieving better 


TABLE I 

Computation TIMES comparison 


Proposed Method 

0.002979 [sec] 

Minimization 

0.015251 [sec] 


BER performance. 

VI. Conclusion 

In this letter, we have proposed a symbol detection method 
for faster-than-Nyquist singling system by SOAV with EISTA. 
We have derived the proximity operator in the SOAV opti¬ 
mization for binary symbol detection. We have also shown 
simulation results to illustrate the effectiveness of the proposed 
method. 

Acknowledgment 

This research was supported in part by ISPS KAKENHI 
Grant Numbers 15H02668, 15K14006, 26120521, 15K06064, 
and 15H02252. 

References 

[1] J. E. Mazo, “Faster-than-Nyquist signaling,” Bell System Tech. J., vol. 54, 
no. 8, pp. 1451-1462, 1975. 

[2] F. Rusek and J. B. Anderson, “Constrained capacities for faster-than- 
Nyquist signaling,” IEEE Trans. Inf. Theory, vol. 55, no. 2, pp. 764-775, 
Feb. 2009. 

[3] J. B. Anderson, F. Rusek, and V. Owall, “Faster-than-Nyquist signaling,” 
Proc. IEEE, vol. 101, no. 8, pp. 1817-1830, Aug. 2013. 

[4] H. Nyquist, “Certain topics in telegraph transmission theory,” AIEE 
Trans., vol. 47, no. 2, pp. 617-644, 1928. 

[5] F. M. Han and X. D. Zhang, “Wireless multicarrier digital transmission 
via Weyl-Heisenberg frames over time-frequency dispersive channels,” 
IEEE Trans. Commitn., vol. 57, no. 6, pp. 1721-1733, June 2009. 

[6] W. Kozek and A. F. Molish, “Nonorthogonal pulseshapes for multicanier 
communications in doubly dispersive channels,” IEEE J. Sel. Areas in 
Commun., vol. 16, no. 8, pp. 1579-1589, Oct. 1998. 

[7] G. Matz, D. Schafhuber, K. Grochenig, M. Hartmann, and F. Hlawatsch, 
“Analysis, optimization, and implementation of low-interference wire¬ 
less multicarrier systems,” IEEE Trans. Wireless Commun., vol. 6, no. 5, 
pp. 1921-1931, May 2007. 

[8] T. Strohmer, “Approximation of dual Gabor frames, window decay, and 
wireless communications,” Appl. and Computational Harmonic Anal., 
vol. 11, no. 2, pp. 243-262, 2001. 

[9] J. B. Anderson, A. Prlja, and F. Rusek, “New reduced state space 
BCJR algorithms for the ISI channel,” in Proc. 2009 IEEE Int. Symp. 
Information Theory, 2009, pp. 889-893. 

[10] A. Prlja, J. B. Anderson, and F. Rusek, “Receivers for faster-than- 
Nyquist signaling with and without turbo equalization,” in Proc. 2008 
IEEE Int. Symp. Information Theory, 2008, pp. 464—468. 

[11] F. M. Han, M. Jin, and H. X. Zou, “Binary symbol recovery via 
minimization in faster-than-Nyquist signaling systems,” IEEE Trans. 
Signal Process., vol. 62, no. 20, pp. 5282-5293, Oct. 2014. 

[12] M. Nagahara, “Discrete signal reconstruction by sum of absolute values,” 
IEEE Signal Process. Lett., vol. 22, no. 10, pp. 1575-1579, Oct. 2015, 
[Online]. Available: http://arxiv.org/pdf/1503.05299vl.pdf. 

[13] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding 
algorithm for linear inverse problems,” SIAM J. Imaging sciences, vol. 2, 
no. 1, pp. 183-202, 2009. 

[14] F. M. Han and X. D. Zhan, “Asymptotic spectral efficiency of digital 
transmission via overcomplete frames with discrete, finite, and uniform 
alphabets,” IEEE Trans. Wireless Commun., vol. 13, no. 7, pp. 3715- 
3725, July 2014. 

[15] N. Parikh and S. Boyd, “Proximal algorithms,” Foundations and Trends 
in Optimization, vol. 1, no. 3, pp. 123-231, 2013. 

[16] H. H. Bauschke, R. S. Burachik, P. L. Combettes, V. E. D. R. Luke, and 
H. Wolkowicz, Fixed-Point Algorithms for Inverse Problems in Science 
and Engineering. Springer, 2011. 








